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Integrating Technology in the Classroom: 
Process Evaluation, Strategies and Issues 



As we approach the 21st century, computer technology, multi-media and electronic networks 
are being integrated in the classroom throughout the country. Today's technology offers 
powerful tools for transforming the way we operate our schools. 

In 1990, the Florida Legislature established the Instructional Technology Grant Program for 
implementing state-of-the-art instructional technology in the classroom. Instructional 
technology was defined as the application of electronic media to the learning process. 

Leon County Schools, a middle size school district in north Florida, was awarded 
approximately $300,000 to introduce and expand the use of instructional technology in the 
classroom. Specifically, various combinations of electronic media (hardware and software) 
were introduced in the classroom and used to prepare, motivate and teach a variety of 
curricula. 

Purpose 

The purpose of this paper is to present the evaluation results of the Leon County Schools' 
"Exploring With Technology" grant funded during the 1992-1993 school year by the Florida 
Department of Education. The grant proposed to integrate a variety of technological 
applications with Social Studies curriculum. 

The main focus of the paper will be on the process evaluation of the implementation. The 
evaluation of project outcomes in terms of student achievement was considered preliminary 
because of delays in equipment deliveries and installations. Funds were granted for one year 
and a complete evaluation report is being finalized. 

Specifically, this paper will: (1) describe the evaluation strategies proposed in the grant, (2) 
briefly review some of the literature related to the evaluation of technologies in the 
classroom, (3) report the evaluation results of the project implementation, (4) briefly review 
the results of the specific outcomes proposed in the grant, and (5) discuss the issues and 
lessons learned during the project's first year of implementation. The lessons are presented 
in the form, of recommended guidelines schools and school districts should incorporate when 
planning to introduce a wide-range of technologies in classrooms. 
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Project Description 

Fifth-grade students in ten elementary schools in Leon, County Florida participated in a year 
long study during the 1992-1993 school year. The Exploring with Technology project 
provided for networking fifth grade classrooms to increase access to telecommunications and 
related media for both teachers and students. In total 1,043 students were served in 36 
classrooms. 

Specific Grant Components 

• Purpose and Vision of the Grant. To empower the students to discover yesterday, 
explore today, and envision tomorrow through the use of technology. The ultimate ' 
purpose was quite ambitious: to better prepare students for a competitive, 
technological world. 

• Curriculum. The exploration thematic unit of the 5th grade Social Studies 
Curriculum and State Student Performance Standards provided the framework for 
developing the fifteen social studies objectives representing three goal areas. 
Beginning with the voyage of Columbus, students gather information from 
telecommunications networks, online information services, video discs and electronic 
mail about early explorers and pioneers. As students progress in their knowledge of 
early explorers and experience with communications technologies, their studies 
progress to today's world. As students make connections among the factors of 
communication, exploration, technology and change, they will use their enhanced 
higher order thinking skills to project how these factors come into play in the world 
of their future. 

• Objectives and Instructional Strategies. To accomplish the instructional objectives, 
several suggested instructional strategies and media activities were proposed in the 
grant. Teachers were required to accomplish instruction in some objectives and were 
free to select other objectives if they wished. Appendix 1 displays the list of goals, 
objectives, and instructional strategies and timelines. These form the basis for the ' 
evaluation objectives proposed for the grant. 

• Mentor Teachers. One teacher at each school was designated the mentor teacher to 
provide training and support as well as model effective use of the technology for other 
project teachers. 

• Hardware. Each school was to receive a basic level of technology. Three project 
teachers at each school were to be provided with a computer and printer to keep in 
their Classroom. In addition, each school was to be provided with a multimedia cart 
with videodisc player, monitor, CD-Rom, LCD overhead panel, scanner and a 
computer with desktop publishing capabilities. The media centers were to be 
provided with a fax machine, a net modem and a CD-Rom Drive. 
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• Networks. Project classrooms in the same school were to be networked. Four 
schools already had file servers and Netway gateways prior to the grant. Four 
schools were to receive the complete Novell network with the MAC NLM and two 
schools were to receive the Novell network without the MAC NLM. 

• Communications. All project classrooms were to be equipped with telephone lines 
and free access to online services - such as FIRN, InterNet, and electronic mail. In 
addition schools were provided funds to access commercial services such as Prodigy 
and CompuServe. 

• Software. Schools were to receive funds to purchase software for communications, 
hypermedia and word processing as well as laser discs and compact disks. 

• Training. All teachers were to receive at least 44 hours of training in the project 
technologies. Mentor teachers were to receive an additional nine hours. 

• Personnel. Funds were granted to provide a project manager to administer the 
acquisition, distribution, and installation of various technology and curriculum 
materials, assist with planning and scheduling training sessions, and facilitate project 
sharing. In addition, a part-time evaluator was dedicated to this project. No funds 
were allocated to provide for technicians dedicated to the project. 

• Evaluation. The main focus of the evaluation was to document and monitor project 
implementation through site visits. Because of uncertainty about what outcomes 
should be expected, behavioral objectives were not proposed in the grant. The 
outcomes were formulated in terms of students' products and activities involving the 
use of a variety of media. The evaluation plan is explained in more detail in the next 
section. 

In summary, the project was innovative in that it approaches the usual fifth grade Social 
Studies Curriculum from a multimedia perspective. Some of these media are fairly new, and 
the users themselves have limited experience with them. Furthermore, the instructional 
environment is distinct from the computer-assisted instructional programs in the 1960s and 
1970s and the more recent applications of Integrated Learning Systems (Computer 
Curriculum Corporation (C.C.C.), WICAT I.L.S. system, Jostens, etc.). 

Evaluation of Instructional Technology Programs: Lessons Learned from the Literature 

The majority of evaluation studies involving the use; of media in the classroom compared 
instructional outcomes for various media. Research reviews on using computers in 
instruction (Jamison, Suppes, and Wells, 1974; Kulik and Kulik, 1987; Becker, 1987; 
Robyler, Castine, and King, 1988) have concluded that computer-assisted instructional 
programs in the 1960s and 1970s were generally more effective in raising students' scores on 
standardized achievement tests than alternative approaches. 
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On the other hand, Reeves (1989) contends that if the purpose of evaluation is to provide 
information for decision making that will improve the quality of life, a decade of evaluations 
of interactive multi-media (IMM) did little to promote the use of IMM in higher education or 
any other level. The vast number of these studies have been media comparisons studies 
seeking to find differences between IMM and another method of instruction. The bottom 
line for many of these studies has been "no significant differences" in effectiveness among 
the media under comparison. 

Becker (1987) pointed out that the existing literature about the effectiveness of computer 
instruction is outdated and suffers from methodological problems. He concluded that 
"existing evidence of computer effectiveness is scanty, and existing studies provide little 
guidance for schools to decide how to use computers for instruction". Furthermore, although 
the media comparison literature is extensive, it does not yet address the complexities of a 
program in the public schools such as Exploring With Technology (Atkins & Green, 1993). 

Reeves (1989) stated that instructional technology suffers from a lack of fundamental work at 
the description level and that a multi-faceted approach to evaluation including the conduct of 
intensive case studies (Stake, 1978) should be adopted. 

Far to often program evaluation involving the use of media is limited to an activity that takes 
place at the conclusion of a program and involves the use of experimental and quasi- 
experimental designs to compare program outcomes. While this is certainly important it 
represents only one role of evaluation. If we are to make informed decisions and sound 
judgements about educational programs, especially those programs that incorporate 
technology, we must adopt an evaluation approach emphasizing a rich description 
(Reeves, 1990). 

Atkins & Green, 1993 pointed out that in media-rich environments evaluators face some keen 
challenges in making judgements about program implementation and outcomes. They 
concluded that "factors that make the judgements difficult include freedom and variance in 
the use of the complex combinations of media, variance in the skills of the teachers and 
students, variance in the attributes of the schools' support staff and equipment, and often, 
profound lack of understanding of the capabilities of the new technology being introduced". 

To date there is no body of research or evaluation on the implementation and effectiveness of 
programs using a variety of technologies. We simply do not know what the program is 
supposed to be doing. Program evaluation is needed to help us understand the 
implementation of these programs. 

Evaluation Plan 

The strategy used in the grant combine elements of formative and summative evaluation. 
During the implementation of the project, activities were formative and the evaluation focus 
was on the continuous monitoring of project implementation and on the need for revisions. 
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The purpose of the summative evaluation was to document to the program staff, the Florida 
Department of Education (DOE), and potential funding agencies the level of implementation 
of the project and the results of the evaluation of the specific outcomes included in the grant 
proposal. 

The information in Table 1 presents the evaluation questions and respective data collection 
procedures proposed in the grant. 

The expected outcomes proposed in the grant address students* technological products and 
achievement, student attendance, attitudes of students, teachers and parents and completion of 
training (see Table 2). 

Data Collection Strategies 

1. Site Visits 

The main strategy for data collection of program implementation were the site visits. 

Each of the ten school site projects was visited three times during the year (October, 
February and May) by a team consisting of evaluation and instructional specialists. The 
purpose of the visits was to qualitatively analyze the utilization of the technology, to assess 
the difficulties and successes during the implementation, and to document the effects 
(intended and unintended). Formative evaluation reports were provided as feedback for the 
stakeholders a few weeks after the visits were completed. Some of the areas the team 
focused on were the changes in the classroom environment, integration of the technology in 
the daily instruction, teacher needs for training, roles of tne media specialist, and availability 
and use of equipment. The visits were arranged in advance to accommodate the teachers' 
planning period. Each visit lasted approximately three hours. Following is a brief 
description of instruments used in each visit. 

October Visit 

The purpose of this first visit was to establish some baseline information and to get 
acquainted with the staff at each school. The instruments used were: 

• School Official's Interview Form - This instrument was responded to orally. 

• Media Specialist's Interview Form. This instrument contained the questions for an 
interview but could also be administered as a written survey. 

• Teachers's Survey Form - This instrument asked questions about hardware use in the 
teachers classroom. 
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• Teacher's Interview Form - An interview was conducted orally with all teachers in a 
group. Questions concentrated on feelings toward the project. 



• Captain's Log Review Form - This form was a simple checklist of items teachers 
were to include in their log, 

• Classroom Observation Form - This included a checklist as well as open-ended 
questions for recording classroom instructional characteristics. 

February Visit 

The purpose of this visit was to assess both the current state of the schools in the project and 
the changes which occurred since the first visit. Instruments used in this visit were: 

• Media Specialist's Interview Form-This instrument asked questions pertaining to the 
roles of the media specialist. 

• Interview Questions for Teachers-The set of questions included the use of various 
types of media, objectives completed to this point and some feelings regarding the 
project. 

• Captain's Log Review Form-This was a checklist to document progress toward 
accomplishment of the instructional objectives. 

• Classroom Observation Form-See October visit, 

• Review Form for Students' Portfolios - This was used to record progress in the 
development of portfolios. 

May Visit 

The final visit was the most comprehensive because it was necessary to collect evidence for 
completion of student products for the summative evaluation report. As before, the 
following instruments were used. 

School Official's Interview Form 

Survey Questions for Media Specialists 

Survey Questions for Project Teachers 

Captain' Log Review Form 

Review Form for Students' s Portfolios 

Classroom Observation Form 
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2. Student Achiev ement 



Student achievement was assessed through student applications of technology and % 
standardized test scores. Teachers' logs, students' logs and portfolios were used to document 
the evidence of accomplishment of the products. 

The California Achievement Test (CAT) replaced the Comprehensive Assessment Program 
(CAP) during the Spring of 1 992-93 . Therefore, a longitudinal evaluation of students' 
achievements was not possible. Instead, CAT language subtest scores of 5th grade project 
students were compared with 5th grade students' scores in matched schools using an analysis 
of covariance design . 

3. Student Attendance 

Attendance records for 1991-92 and 1992-93 were analyzed to assess improvement in 
attendance. 

4. Attitudes of Students. Teachers, and Parents 

The Leon County Attitude Survey was utilized to determine attitudes toward selected aspects 
of the project. The 1992-93 responses of project fifth grade students, their parents, and fifth 
grade teachers were -compared with their counterparts in ten matched schools. 

Summary of Findings by Evaluation Question 



Evaluation Questions related to Process 



1. Which technologies (hardware/software) were available in each classroom? 

All the equipment proposed in the grant was ordered and installed in all project 
schools but not within the timelines specified in the grant. Bottlenecks for bringing 
equipment on line caused the completion of hardware installation to be delayed in 
some cases until the end of the year. Major problems included vendor-related 
problems (e.g., back order and incorrect shipments), district/state purchasing 
processes for large technology items, technology decisions required of project 
schools, and a shortage of district technical support personnel. School networks were 
not completed until near the end of the 1992-93 school year primarily because wiring 
was not completed until late in the year. Also installation of classroom telephone 
lines proved to be a problem due to a district moratorium on telephone lines and 
back-logged work. 
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What was the degree of use of the computer workstations, and other media such 
as videodiscs, LCD, overhead panel, CD-ROMS and telecommunications? 

There was ample evidence from the three evaluation visits that both teachers and 
students were primary operators of their computers and video equipment. 
Furthermore, approximately two thirds of the project teachers were applying video 
discs/VCRs to classroom instruction on a weekly basis by February. In contrast, 
classroom applications of telecommunications were made by less than half of the 
project teachers for two reasons. First, there was much to be learned in order to 
operate the various telecommunications equipment, and teachers were expending much 
time learning other basic equipment. Second, telecommunications were not 
operational until the second half of the year. 

Did the use of media increase during the school year? 

Yes. According to self-report inventories, computers were already being applied in 
October in individual or small group settings approximately four hours per week per 
classroom . This was up from a median of less than one hour per week reported for 
the previous year when teachers used computers typically for administrative purposes. 
Stand-alone computers, videodiscs, and VCRs were the most common media 
applications throughout the year. Baseline survey data indicated that many teachers 
had never operated any of the sophisticated media before the project began. All in 
all, considerable progress was made as teachers learned to apply, on a daily basis, 
Macintosh and IBM 386 technology as well as videodiscs. By the end of the year, 
some teachers had also experimented with telecommunications and multimedia 
applications. 

Which parts of the grant lack implementation? 

Overall, teachers focussed on accomplishing applications using their computers and 
videodisc players with individual students or small groups and for large groups with 
projected images. What was not accomplished in the spirit of the grant by most 
teachers during the first year was regular applications of multimedia video, 
telecommunications, and communication through networks. A minority of teachers 
were experimenting with these applications, but most were actually only achieving 
mastery of integrating their computers and videodisc players in their social studies 
curriculum by the end of the year. Also, much of the equipment for multimedia, 
telecommunications, and network communications did not come on-line in teachers' 
classrooms until nearly the end of the year or after the year was completed (i.e., 
video spigot, CD-ROMs, data displays, scanners, networking). The most 
disappointing characteristic of the first year was the failure to bring networks on line. 
This virtually eliminated experimentation with telecommunications because 
telecommunications could only be achieved from within mentor teachers' classrooms 
and/or media centers. Non-mentor teachers and their students were to be able to 
telecommunicate through their networks. 
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How did the technology change the classroom environment and the total school? 

Teachers reported that classroom management and teaching styles by the end of the 
year were heavily influenced by students working independently and in small groups 
with project equipment. However, the price being paid was illustrated by the fact that 
93% reported that lesson preparation time had increased considerably from the 
previous year. 

School administrators indicated that the project was raising the technological 
awareness in their school (44%) and project equipment was being shared with other 
teachers (33%). Furthermore, sixty percent of the media specialists felt that they 
were becoming more familiar with the state-of-the-art technology and their students 
were using more technology. All in all, the project was seen as a catalyst to upgrade 
technology in their schools. 

What were the perceptions/feelings toward the project? 

Teachers' feelings seemed to vary as the year progressed but were always positive 
about the worth of the program. At the end of the year, ratings on a five point scale 
indicated that teachers considered the project worthwhile (4.8) and enjoyable (4.7). 
Furthermore, they viewed the project as much less difficult to implement than they 
did earlier in the year. In fact, nearly all of the project teachers and principals 
expressed positive feelings about teachers' professional growth and their gain in 
technological empowerment during the school year. Almost all desired to continue 
the project during the 1993-94 school year. 

Some frustration and anxiety were also evident throughout most of the year mainly 
because of the uncertain timelines for arrival of the equipment and uncertainty about 
their own abilities to deal with specific applications. 

Were the teachers being able to complete the goals/objectives of the grant? 

Results of the evaluation visits indicate that project teachers were able to accomplish 
the majority of their instructional objectives but not necessarlLy using the suggested 
media activities nor within the timelines specified in the grant proposal. Because of 
delays in equipment arrivals, the project actually began several months behind the 
schedule listed in the grant proposal. As a group, project teachers completed 
approximately 66% of their preselected objectives by the May visit. Approximately 
two-thirds of the incomplete objectives were accounted for by instructional strategies 
requiring the use of telecommunications or multimedia equipment. (See Table 3 for a 
listing of school-selected objectives). Furthermore, some teachers at every school 
completed instructional objectives other than those which were pre-selected by their 
school. 
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8. Was the amount of training sufficient to incorporate appropriately the technology 
in the classroom? 

Teachers received at least the 44 hours of training stated in the grant, although the 
specific topics were changed to reflect the needs of the teachers. Much more training 
was conducted in computing basics and curriculum enhancement than was stated in 
the grant proposal, and less training was conducted on the more advanced topics. 
The issue concerning training was not necessarily the amount, although it is never 
enough, but the timing of training. Sometimes training occurred well before the 
arrival of equipment. By the time the equipment arrived teachers had often already 
forgotten what they learned. Nevertheless, during interviews, teachers felt positive 
about the training provided to them by the grant but believed that more follow-up 
training was needed. 

E valuation Questions related to Outcomes 

9. To what extent were the student outcomes achieved? 

Many classroom applications of technology were accomplished by project teachers 
and students even though the evaluation of product outcomes yielded mixed results. 
The project definitely achieved the expected outcomes for two products involving E- 
Mail and/or Fax (See Table 2, outcomes 3 and 6). Two more product outcomes 
involving Timeliner software and posters (2 and 5) might have been achieved, 
however, this was not truly determined because the quality of teachers record-keeping 
for student products varied considerably. (In fact, thirteen of the project teachers 
simply did not maintain portfolios, therefore the scope of students' products were not 
accurately documented.) The project did not achieve its expected outcome of students 
contributing work to a graphic slide show. The activities associated with this outcome 
required the use of more complex technology and probably required more time than 
teachers were able to assign to them. 

The evaluation yielded no evidence that language skills of students improved as a 
result of the project. Comparisons of project students' CAP subtest scores with those 
of students of matched schools yielded no significant differences in language arts 
achievement when adjusted by a covariate of prior year CAP language scores. These 
results conform with the literature in the field. Student achievement as measured by 
standardized tests may be an important measure of the success of an educational 
program but improvement in test scores may take multiple academic years to occur. 
In addition, many standardized tests are not yet sensitive to the skills in which 
technology develops. An achievement test may not reflect improvements in 
achievement when they in fact exist. 
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Was there a change in the level of student involvement in the school as measured by attendance 
data? 

The project did not improve the attendance rate of th£ students as a whole. The mean absentee rate 
was approximately 3.6% for both the 1992-93 project year and the prior year. This translated into less 
than 6.5 days per 180 day period. The question is formulated whether any project should be expected 
to generate enough excitement in elementary students to significantly decrease an absentee rate which is 
so small? 

Are the attitudes of students, parents and teachers more positive than their coi .iterparts in 
matched schools? 

Overall, the data did not indicate project students' attitudes were more positive than their peers in 
matched schools as measured by the Leon County School Attitude Survey. The exception was media 
and library services where the mean project student's response was 10% more positive. Most of the 
project parents responses showed a more favorable attitude toward their schools overall than those of 
matched-group parents. 

Project teachers* mean responses for the four questions evaluated were higher than those of matched 
teachers. For media and library services, the difference was found to be more than 10%. 

Caution is advised when considering these results. The survey was not especially designed to assess 
attitudes of project stakeholders. Furthermore, comparisons of project schools responses were made 
with those of matched schools without the benefit of a covariate to adjust for initial group differences 
because data was not collected which allowed grade levels to be identified, 

Overall, feedback from teachers and principals indicated that substantial positive feelings were created 
about technology and its worth to education in the project schools (see question 6). 

What were the m^jor benefits associated with the project? 

Some of the highlights are: 

• Almost all project teachers and most students learned to use and frequently applied later 
generation Macintosh and DOS desktop computers to classroom instruction. 

• Approximately two-thirds of the project teachers were applying video discs/VCRs to classroom 
instruction on a weekly basis by February. A minority of project teachers began 
experimenting with telecommunications applications during the second semester. 

• The technology put into place exceeded the existing technology in both quantity per classroom 
and quality. 

• The district staff and ten schools learned much and collected much information about 
equipment availability, compatibility and the resources and coordination required to bring 
schools on line. 

• Substantial positive feelings were created about technology and its worth to education. 
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• The project served as a catalyst for teachers to work with each other, district staff, their school 
administrators, and their media specialists to solve technical problems and plan social studies 
lessons. 

• The schools gained valuable experience in the implementation of multiple technologies in the 
classroom, experience that is worth sharing. The sharing of information and experiences can 
help staff try new approaches to teaching. 

• Most project teachers received more media training than had been received by most other 
teachers in their schools, thereby, providing their school with a larger nucleus for preparing 
technology plans and training other teachers. 

• Overall, the project conveyed the belief that technology can be used to improve classroom 
instruction but technology alone is insufficient to accomplish this. Many other changes will be 
necessary, including a fundamental change in instruction and learning methods. 

Conclusions and Recommendations 

In reviewing the above findings, it is evident that the scope of the technological applications proposed for the 
project classrooms was excessive given the timelines. The implementation of the proposed plan might be best 
characterized as an attempt to pack a multi-year project into one year. Formative evaluation reports revealed 
holdups in equipment delivery which caused the completion of hardware installation to be delayed until the end 
of the year and subsequently affected the full implementation of the project. Also training sometimes occurred 
well after or well before equipment was operational in the schools, causing additional frustrations. 

The test score data did not show major difference in achievement. This was hardly surprising since the 
relationship between the skills promoted by the technology and the tests utilized were unclear, and further, 
because improvement in test scores may take multiple academic years to occur. Fortunately, the students* 
products and portfolios yielded rich data although they proved to be difficult to evaluate. 

Despite the difficulties experienced throughout the first year, there was abundant evidence of major 
accomplishments, and schools, teachers, and students appeared to have benefitted significantly. Reports by 
school staff revealed that by the end of the year teachers were just reaching the stage where they were prepared 
to implement the project in the spirit of the grant. Although the project received no direct second year funding 
from the DOE, a sirjall amount of grant and other resources were granted by the district for teacher training and 
evaluation. 

In light of the foregoing, the following recommendations are offered. 

• Media-rich programs should be envisioned as multi-year projects with modest goals for 
implementing equipment and training in an orderly fashion during the first year. Experience from 
this project indicates that modest amounts of media should be put into classrooms in an orderly process 
beginning with computers and videodiscs. Other media should be added when most of the teachers 
have been trained and have demonstrated their ability to apply preliminary equipment. 

• Funding agencies should consider supporting three year studies of technology 
projects. The first two years should focus on process variables and the third year 
should continue documentation of the process as well as assessments of the 
effectiveness of the project according to a variety of indicators. 
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Expected outcomes for project teachers and students should be stated in terms of 
empowerment and should be evaluated in a systematic manner that is not totally 
dependent on teachers' logs and students' portfolios and products* Expected 
outcomes should be stated in terms of teacher and student empowerment. 
Empowerment implies not only what skills have been performed, but, also, what the 
subjects are enabled to do as a result of having learned the skills. 

Technicians representing all pertinent areas and who are knowledgeable with the 
physical attributes and equipment of the targeted facilities should be consulted on 
a regular basis in planning the technology designs and timelines of the project* 

In the post project debriefing of the project technicians and manager, the technicians 
pointed out that most of the technical bottlenecks could have been either planned for 
or avoided if technicians had been more involved and consulted during the writing of 
the grant. 

Training in the utilization of software and hardware should be "just in time". If 
possible, it should be delayed until the equipment is in place. As previously 
indicated, the timing of the training was crucial for the appropriate utilization of the 
equipment. The lag between training and complete hardware installation diminished 
the value of training. 

Funding agencies should ensure a genuine commitment to effective evaluation so 
that sound decisions can be made about the program being evaluated. One of the 

positive aspects of this project has been the evaluation team's ongoing support of the 
project. Formative evaluation should begin when program planning begins and should 
continue through program implementation until it is time for summative evaluation. 

More detailed investigations are needed related to whether educational technology 
has a significant role to play in bringing substantial improvements to both 
teaching and learning. 
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